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1.1 Introduction 


•Why multi-level models? 

• Avoid aggregation bias and inflated std. err. 

• Disentangle group- and individual-level effects 

• Strengthen generalizability to group-level pop. 

• Model changes over time 
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1.2 Definition of Terms 


SAT 


• College admissions test consisting of Critical 
Reading, Math & Writing sections on 200-800 scale 

Advanced Placement 

• High school program created to give students the 
opportunity to take college-level courses 

Visit: http://www. c ollege h o a r d .co m / 
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1.3 Multi-level Modeling 
Opportunities 

• Model first-year grade point average 
(FYGPA) at 1 1 0 colleges and universities 

• Model prob. of participation in Advanced 
Placement Program® at many high schools 

• Model FYGPA with high-school- and 
college-variability with cross-classified data 
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1.4 SAS Requirements for 
Multi-level Modeling 


• Many observations 

• Large sample size within each group 

• Sufficient number of groups 

• Computing power 

• Procedures (and optional statements within 
some procedures) are memory-intensive 


• Data structure 
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1.5 Nested Data Structure in SAS 
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1.6 Repeated Measures Data in SAS 



8 Multi-level, NYASUG, Dec. 2008 


n 


CollegeBoard 


connect to college success' 
www.coUegeboaid.com 





1.7 SAS Mixed Effects Procedures 


• Main procedures discussed: 

• PROC MIXED 

• PROC GLIMMIX (experimental in v9.1.3; v9.2) 

• Others with mixed model capabilities: 

• PROC GLM; PROC HPMIXED (v9.2); PROC 
LATTICE; PROC NESTED; PROC NLMIXED; 
PROC VARCOMP 
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2.1 From Linear to 

Multi-level Models 

• Multi-level models (MMs) as generalization 
of linear and generalized linear models 

• Assumption of independent error terms 


Ytj - Pq+ fikXtjk + r i) with 


PROC REG DATA= DATA_001; PROC MIXED DATA= DATA_001 

MODEL Y = XI -Xp; * ► MODEL Y = XI -Xp; 

RUN; RUN; 
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2.2 Null Multi-level Model 


•Two-level data; individuals within groups 

• Intercepts vary by group; no other pred. 

• Notation from Raudenbush & Bryk (2001). 


Model 

= Poi + r l 

= 7oo 


Pq j 


Variance Components 

0 ,< 7 2 ) T 
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SAS Code 

PROC MIXED DATA= DATA_001 
COVTEST ; 

ODEL Y = / SOLUTION; 
RANDOM INTERCEPT / 

'= Group TYPE= VC 


RUN; 
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2.3 Null MM as a Single Equation 


• Substituting the expression for ft j from the 
group-level into the individual-level 
equation: 
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2.4 Graph of Null MM 
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2.5 Do we Need a Multi-level Model? 


• Based on the null model (or an ANOVA 
table) we compute the intra-class 
correlation coef. (ICC) for linear models 

• Indicates the proportion of error variance in the 
outcome relative to overall error variance. 

_ tqq _ Group-level Error Var. 

^ ~ T 00 + <7 2 ~ Total Error Var. 
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2.6 SAS Output: 
Null Model 

• ICC= 0.014/0.10 

= 0.14 

• ICC not 
meaningful for 
GLMM 

• COVTEST option 
in PROC MIXED 

CollegeBoard 
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Iteration History 

Iteration 

Evaluations 

-2 Res Log Like 

Criterion 

0 

i 

144.96738154 


1 

2 

135.02577864 

0.00003089 

X 

1 

135.02116407 

0.00000005 

3 

1 

135.02115700 

0.00000000 


Convergence criteria met. 



Covariance Parameter Estimates 

Cov Farm 

Subject 

Estimate 

Standard Error 

Z Value 

PrZ 

Intercept 

Group 

0.01422 

0.007427 

1.91 

0.0278 

Residual 


0.08777 

0.008638 

10.16 

< 0001 


Done 


> /a 
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2.7 SAS Output: Null Model 


• Random intercept specified 

• No other predictors included in the model 


16 



Multi-level, NYASUG, Dec. 2008 


n 


CollegeBoard 


connect to college success™ 
www.coUegeboaid.com 


3.1 MM with Level- 1 Random 
Intercepts & Fixed Slopes 

• Intercepts vary by group; other predictors 
are fixed 


Model 

n, = fa, ^ „ 

fioj = Yoq + Wo/ 
fikj — VkQ 

Variance Components 

r ir N(Q, (7 2 ) T = [ Too ] 


SAS Code 
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OC MIXED DATA= DATA001 
EST; 

/ SOLUTION; 
RANDOM INTERCEPT / 

SUB JECT= Group TYPE= VC 


RUN; 
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3.2 Graph of MM with Level- 1 
Fixed XI Slope 
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3.3 Graph of Null MM 
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3.4 SAS Output: 

Fixed Slopes 

• Same random 
effects as null 

• Slopes are 
constrained to be 
equal across 
groups 
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3.5 MM with Fixed- & Random- 
Intercept and Slope Effects 


Intercepts and other parameters vary 


Model 

Z P 

Pkj^k} + r tj 


Pq} - Fco 

fik] = YkO 


+ 


*0; 

u kJ 


SAS Code 

PROC MIXED DATA= DATA001 
COVTEST ; 


Variance Compon^pfe- ^ model y= xi-xp / solution,- 


rtj~N{ 0, o -2 ) T = 


Tqo 0 0 

0 x 0 

0 0 T, 


RAND£n<INTERCEPT Xl-Xp>/ 


!UBJECT= Group TYPE= VC 


RUN; 
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3.6 Graph of MM with Level- 1 
Fixed- and Random-Effects 
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3.7 SAS Output: 
Random Slopes 


• Same random 
effects as null 

• Slope for XI 
allowed to be vary 
across groups 

• Notice Tu in 
CovParms table 
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Covariance Parameter Estimates 

C o v Farm 

Subject 

Estimate 

Standard Error 

Z Value 

PrZ 

Intercept 

Group 

17.9842 

3.9742 

4.53 

< 0001 

XI 

Group 

1.1365 

0.4328 

2.63 

0.0043 

Residual 


87.1285 

2.9809 

29.23 

< 0001 


Solution for Fixed Effects 

Effect 

Estimate 

Standard Eitoi 

I>F 

t Value 

Pr> |t| 

Intercept 

3.9956 

1.2173 

99 

3.28 

0.0014 

XI 

3.3104 

0.2124 

99 

15.58 

< 0001 

X2 

0.01529 

0.001597 

1758 

9.57 

< 0001 

X3 

0.01469 

0.001621 

1758 

9.06 

< 0001 

X4 

0.01672 

0.001573 

1758 

10.63 

< 0001 




Done 
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3.8 Generalized Linear 
Multi-level Models 


•With a non-normal outcome, it may be 
appropriate to use a generalized linear 
multi-level model (GLMM) 


• PROC GLIMMIX will meet most needs; PROC 
NLMIXED may be appropriate otherwise 

• Recall ICC not meaningful; rely on size and 
standard error of cov. parameter estimates 
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3.9 SAS Specification of GLMM 


PROC GLIMMIX DATA = Data_004 
METHOD = RSPL INITGLM IC= PQ 
NAMELEN= 65 NOCLPRINT ; 

CLASS GroupID; 

NLOPTIONS TECHNIQUE= NRRIDG; 

MODEL Events XI -Xp / DIST= BINARY 
LINK= LOGIT SOLUTION CL; 

RANDOM INTERCEPT / SUBJECT= GroupID; 
RUN; 
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3.10 PROC GLIMMIX for Propensity 
Score Matching 

•Add the following to our PROC GLIMMIX: 

OUTPUT OUT= Data_004_Pred 

PREDICTED (BLUP ILINK) = Treat_Prob_Pred 
STDERR (BLUP ILINK) = Treat Prob SE; 


• Use the predicted probabilities to perform 
propensity score matching. 
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3.11 Notes on Covariance Structure 

•Theory and subject matter expertise should 
dictate the covariance structure 

• For hierarchically structured data: 

• Generally most restrictive (Jparm.): type= vc 

• Generally least restrictive ( J 2 parm.): type= un 

• Many other options; check OnlineDoc 
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3.12 What Can Go Wrong? 


• Convergence problems 

• PROC HPMIXED? Need SAS 9.2. 

• Another estimation method? 

• Fewer fixed- and random-effects? 

•Variance components not estimated 

• Consider modifying RANDOM statement. 
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3.13 What Else Can Go Wrong? 


• "Out of memory" Error 

• Some mixed procedures are prone to this error 

• "PROC MIXED is looking for a contiguous 
memory space and cannot find one large 
enough due to the relocation of the operating 
system modules." -SAS Problem Note 15060. 

• Solution: Try to defragment the target drive. 


29 


a 


CollegeBoard 


Multi-level, NYASUG, Dec. 2008 

connect to college success"’ 
www.coUegeboard.com 


3.14 Diagnosing and Solving 
Problems 

• Diagnose what's going wrong and when 

• Use the LOGNOTE option on PROC MIXED for 
estimation progress step-by-step 

• Remove statements / options to find the issue 

• If convergence is a problem, consider re- 
parameterizing the model 

• Especially in the case of generalized linear MMs 
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3.15 Example of LOGNOTE Output 


NOTE: Levelizing effects. 

NOTE: Processing subject and group effects. 
NOTE: Setting up data. 

NOTE: Loading data. 

NOTE: Computing likelihood in iteration 0. 

NOTE: Computing G derivatives in iteration 

NOTE: Computing likelihood in iteration 1. 

NOTE: Computing G derivatives in iteration 

NOTE: Convergence criteria met but final 
hessian is not positive definite. 

NOTE: Computing likelihood in iteration 2. 


NOTE: A linear combination of covariance 
parameters is confounded with the residual 
variance. 

NOTE: Computing Cholesky root of cross- 
products matrix. 

NOTE: Computing H matrix. 

1. NOTE: Computing Type 3 sums of squares. 

NOTE: PROCEDURE MIXED used (Total 
1 process time): 

real time 0.35 seconds 

cpu time 0.14 seconds 


31 


Multi-level, NYASUG, Dec. 2008 


n 


CollegeBoard 


connect to college success™ 
www.coUegeboard.com 


4.1 Research & SAS Applications of 
Multi-level Models 

• Model first-year grade point average at 1 10 
colleges and universities 

• Model probability participation in Advanced 
Placement Program® at many high schools 

• Model high-school- and college-variability in 
the effect of AP® participation 
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4.2 Model FYGPA Across 110 
Colleges 


• Outco m e: FYGPA, as normal continuous 


•Levels: student and college (nested) 
• P o ssi b le S A S Pro ce d ures: 
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• PROC MIXED 

• PROC HPMIXED (new in 9.2) 

• For large number of fixed- or random-effects 
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4.3 Model AP Participation Across 
Many High Schools 

• Outcome: Indicator of AP Participation 
•Levels: student and high school (nested) 

• High schools may be nested within districts 

• P o ssible v S A S Pro cedures: 

• PROC GLIMMIX (9.2; exp. in 9.1 .3) 

• PROC NLMIXED 
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4.4 Model Effect of AP® 
Participation across HS & Colleges 


• Outcome: FYGPA, as normal continuous 
•Levels: student, high school and college 

• Data are cross-classified by HS and college 

• P o ssible v S A S Pro cedures: 

• PROC MIXED 

• PROC HPMIXED (new in 9.2) 

35 Multi-level, NYASUG, Dec. 2008 

C'b connect to college success™ 

f CollegeBoard www.collegeboard.oom 



4.5 HS & College Cross-classified 
Data 


• Data are not strictly nested 

• Students from high schools A, B and C attend 
colleges I and II 


• Random effects: high school and college levels 
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4.6 SAS Specification of CCMM 
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PROC MIXED DATA = Data_002 COVTEST; 
CLASS HSID CollegelD; 

MODEL Y= XI -Xp / SOLUTION CL; 
RANDOM INTERCEPT XI / 

SUBJECT= HSID; 

RANDOM INTERCEPT XI -Xp / 


SUBJECT= College ID; 


RUN; 
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4.7 Non-Educational Applications 


• Multi-site clinical trials 

• Patients strictly nested within sites or cross- 
classified across sites 

• Retail applications 

• PROC GLIMMIX to estimate same-store sales 
over quarterly observations 
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4.8 General References 


• Multi-level Modeling Texts: 

• Raudenbush, S. W. and Bryk, A. S. 2002. 
Hierarchical Linear Models: Applications and 
Data Analysis Methods. Second Edition. 

• Snijders, T. A. B. and Bosker, R. J. 1999. 
Multilevel Analysis: An Introduction to Basic 
and Advanced Multilevel Modeling. 
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4.9 SAS-Oriented Reference 


• Multi-level Modeling Articles for SAS: 

• Singer, J. D. 1998. "Using SAS PROC MIXED 
to Fit Multilevel Models, Hierarchical Models, 
and Individual Growth Models." Journal of 
Educational and Behavioral Statistics, Vol. 24, 
323-355. 
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5.1 Question / Answer Period 


•Any questions? Statistical, SAS-oriented or 
otherwise? 
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Thank You 


• Thank you all very much for having me. 

• Special thanks to Martin Feuerman of 
NYASUG and Mary-Margaret Kerns of The 
College Board. 
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Contact Information 


Brian F. Patterson 

The College Board, Research & Development 
Assistant Research Scientist 

212 - 713-7714 

bpatterson@collegeboard.org 

http://www.collegeboard.com/research 
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